Hyperspacings and the Estimation of Information Theoretic Quantities

The estimation of probability densities from data is widely used as an intermediate step in the estimation of entropy, Kullback-Leibler (KL) divergence, and mutual information, and for statistical tasks such as hypothesis testing. We propose an alternative to density estimation– partitioning a space into regions whose approximate probability mass is known–that can be used for the same purposes. We call these regions hyperspacings, a generalization of spacings in one dimension. After discussing one-dimensional spacings estimates of entropy and KLdivergence, we show how hyperspacings can be used to estimate these quantities (and mutual information) in higher dimensions. Our approach outperforms certain widely used estimators based on intermediate density estimates. Using similar ideas, we also present a new distributionfree hypothesis test for distributional equivalence that compares favorably with the Kolmogorov-Smirnov test. Using hyperspacings, it is easily extended to multiple dimensions.